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ABSTRACT 

Research vas conducted in 1972 to evaluate a program 
designed to improve the science teaching in grades seven, eight and 
nine in the Province of Saskatchewan* Since the program included a 
strategy for implementation and the use of an instructional package, 
a two-part evaluation vas conducted. A general survey was used to 
evaluate the implementation phase and an in-depth study performed on 
a sample of classrooms to evaluate the effectiveness of the 
curriculum packages. The results have implications for curriculum 
change, in addition to assessing modified products from the 
curriculum reform movement. (Author) 
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Introduction 

The purpose of this study uias to conduct an av/alua- 
tlon of a program designed to improv/e the science teaching 
at the junior high school level in the Province of Saskatchewan. 
Since the program involved a strategy for implementation as 
veil as the use of an instructional package, a tuio part evalu«* 
atidn «as conducted* 

The first part focused on the implementation phase of 
the program and involved the use of an instrument adminis- 
tered to all teachers in the province. The second part 
involved classr*oom visits and the use of student measures and 
observation schedules • 

In this paper the background! procedures and results 
of the study are outlined briefly to provide a basis for the 
thoughts expressed in the concluding section of the paper 
concerning curriculum evaluation and curriculum change. 
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Background of the Study 

3y 1970 science 0ducatlon had undergone two decades 
of curriculum reform both in Britain and North America, 
Attention had focused on major curriculum projects designed 
to improve the teaching of science in the schools. The 
period had emphasized innovation, development and implementa- 
tion* The developments associated uuith this wave of cur- 
riculum reform influenced the science programs organized for 
grades 7, 8 and 9 in Saskatchewjan • 3y the early 1970's, a 
plateau had been reached in program development for these 
grades in the province which offered opportunity for conduct- 
ing an evaluation of the programs in use. 

It was evident in 1972 that new thinking related to 
science teaching was creating pressures for change. The 
emphasis on environmental education and the concerns being 
expressed for a more humanistic^ approach to science teaching 
reflected a change in thinking from the previous emphasis on 
the disciplines of science. Furthert the programs developed 
in the 60*s were being criticized on several accounts." Such 



For an example in the current literature , see 
Robert Samples, "Science - A Human Enterprise," The Science 
Teacher , 1972, 39, 26-29. 
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In a talk delivered to the National Association of 
Biology Teachers on October 24, 1970, in Denver, Colorado, 
Paul de Hart Hurd proposed that the inquiry techniques 
developed in the 60*s are not sufficient for problems facing 
students in the 70*s» 
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thinking mas suvely to produce a r8cori<5 truct ion of science 
curriculum if not a neiii wave of curriculum reform in the 
70*St To assure that these future devGlopmen ts be built on 
information rather than speculation, it luas felt that an 
assessment of past attempts at curriculum change in the 
province was necessary. 

The emphasis on curriculum evaluation and account- 
ability so evident in the late 60's pi-^i^vided an additional 
backdrop to the study* The notion of accountability had 
reached the province by the early 70's as board members were 
beginning to raise the question of value for their dollar. 
Since the Division III science program involved considerable 
expense for implementation it seemed appropriate that some 
means of realting dollar value to implementation should be 
attempted. 

Purpose of the Study 

The study uuas designed to assess the degree of 
implementation of the program and the amount of change it 
had accomplished in the classroom. The specific objectives 
of the two part evaluation study mere as follows s 

(1) To assess the extent to which local school districts 
had adopted the instructional package and conducted the 
appropriate in-service education programs. 

(2) To determine if factors such as cost, demographic 
variables of teachers and school organization were related to 
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program implemsntatlon and use« 

(3) To construct a teacher profile of demographic 
characteristics and school profile of organizational factors 
typical of high and low implementation. 

(4) To determine if the use of the instructional package 
could be related to the interests, attitudest classroom 
perceptions and cognitive ability of the students and the 
instructional mode of the teacher* 

The Division III Science Program 

Development of the Program 

The Department of Education established several com- 
mittees in 1964 for the purpose of revising all courses of 
study at the junior high school level in the Province of 
Saskatchewan • The science area uias to operate within a 
general philosophy especially established for Division III 
(grades 7, 8, and 9). 

The science committee consisted of four teacher 
representatives, a principal, a superintendent, a representa 
tive from the Faculties of Education , and one From the Depart 
ment of Education. Its charge uias to revise or develop 
courses of study for grades 7, 8, and 9 science teaching. 

The initial deliberations of the committee gave rise 
to two options for accomplishing their task. First, since 
the grades 7 and 8 courses had been revised only three or 



four years earlier perhaps only minor revisions mere needed 
for these gradesv allouilng a major effort to be spent on a 
grade 9 program. Alternatl\/ely i the committee could start 
afresh without reference to previous programs. 

The committee elected the latter alternative for 
several reasons. First, the feedback from teachers sug- 
gested that the grade 7 and 8 programs mere not being uiell 
received. Secondly! science education uias undergoing major 
changes-^developments had occurred mhich Had not reached 
fruition prior to the 1961 revision of the seventh and 
eighth grade programs. 

The committee held regular meetings in 1964-65 and 
concentrated their efforts during a tuio meek mriting con- 
ference in the summer of 1965. 

It soon became evident, during the summer uiriting 
conference of 1965 that the committee would not be able to 
go beyond setting up a structure for the science program and 
uiriting an introduction. The latter was to include a philo- 
sophy, objectives, and a list of skills. 

A program which was being developed in the United 

3 

States, the Earth Science Curriculum Project, seemed to 
possess many of the characteristics the Committee intended 



Hereafter referred to as ESCP. 
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for the grade 8 earth science program. Schools in Saskatoon 
and Yorkton piloted the program during the 1966-57 school 
year. 

During the next year 19S6-67, three classrooms in 
Saskatoon made use of a v/ariety of experiences on a trial 
basis with the expectation, that these might be included in a 
grade 7 program. The experiences were drawn from a variety 
of sources and many were developed by the teachers involved. 
They were organized around conceptual schemes somewhat 
similar to those in BoCS. 

In the summer of 1967 a second writing conference was 

held in order to further develop the programs for the three 

grades. Separate teams worked on the life science program 

for grade 7, the earth science program for grade 8 and the 

physical space science materials for grade 9. Further work 

was done on the philosophy of the program and its conceptual 

structure. The seventh grade team wrote investigations based 

on the past year's trial. The earth science materials were 

revised somewhat to provide alternatives to the ESCP text 

which had been used the previous year. A group began work 

on the space science program attempting to draw together 

the diverse thinking about the area. The result of this work 

5 

was a tentative course outline for the three grades. 



BSCS is the acronym for Biological Science Curriculum 

Study. 

This document was entitled the Tentative Course Out- 

O . line Division III S cience , September 8, 196^ . 
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The program was implGmenled in two stages. The first 
involved the spreading of pilot projects. In this phase 
in-serv/ice education was accomplished through teacher-inter-* 
action and co-operationo The second stage was more wide- 
spread and took place ^'through half dr.,y ''jorkshops held 
throughout the province. The workshops luere led by teachers 
trained by the Division III Science Committee during a two 
day workshop held in Saskatoon in August, 1968. Since this 
date in-service education has largely been the perogative of 
local school districts • 

Structure and Objectives * The general aspects of 
the program were to include the following characteristics t 
1 • The program should have a unitary flavor for the 

grades and provide a three-yaar cycle * It was hoped 
that the grade 9 year would provide a capping effect 
upon the previous two years and that a student would 
touch upon the main content areas of life, earth and 
space science during his grade 7, 8 and 9 school 
career. The committee wished to avoid having three 
separate courses for the three grades. 
2 • The program should be built around conceptual and 

behavioral schemes in Science . Recent thinking in 
science education at that time seemed to suggest 
that a more fruitful way of organizing curricula was 
to utilize major ideas in scienpe rather than subject 
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matter content. In additionj certain behavioral 
themes uiere identified u/hich utould serv/e to gi\/e 
coherence to student experience* These themes were 
to provide an overall structure to which the three 
content areas ufould contribute • Specific concepts 
within each subject area were identified for attain- 
ment at the three grade levels. 
3. The Science program should emphasize student involve- 

ment with materials and the investigations should 
provide for an inquiry approach to learning . The 
implications of this facet of the program were that 
equipment and materials should be available. It was 
noted that the inquiry and investigation approach 
would require a major change in role for many 
teJachers. 

As it was noted in the previous section, the develop- 
ment of the program took place over a period of three years. 
At the point where the program was considered ready for 
adoption the seventh and eighth grade were more thoroughly 
developed than the ninth grade space program. Difficulties 
had arisen frequently on attempts to define an area of space 
science* This controversy found itself into implementation 
of the program as many schools adopted the Introductory 
Physical Science^ program for grade nine rather than the 



Hereafter referred to as IPS, 



space science program* 

The objectives of the program were the follouiingj 
!• To dev/elop in the student an appreciation for, and an 

interest in scienc8# 

2. To challenge the student to think and reason through 
scientific study. 

3. To foster a spirit of inquiry. 

4. To develop an understanding and appreciation of the 
methods by which scientists discover neiu knowledge. 

5. To develop a deeper insight into the inter-relation- 
ships that exist in science • 

6* To develop and broaden the student's understanding of 

the big ideas of science. 
7. To help the student acquire some of the skills and 

processes of science. 

The Evaluation POodel 

The difficulties associated with performing an 

evaluation after a program has been in operation for a number 

of years stem largely from the fact that little has been 

written concerning appropriate means for conducting ex^ post 

n 

facto Bvaluations. The Campbell and Stnnley classification 
of research designs is largely inappropriate. The recent 



'^See D.T, Campbell and J.C. Stanley, Experimental 
and Quasi^experimental Design for Research , Rand WcMally, 
1963. ■ 
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models developed f.07" curriculum ©valuation provided a base 
to work from but were also somewhat inappropriate since they 
were designed to follow programs through the planning, 
development implementation and final evaluation stages. 
Hence only partial use could be made of them. The situation 
was further complicated because there was no baseline data to 
work from and no steps were taken as the program was intro- 
duced that might facilitate evaluation at a later data. 

The model developed for this study utilized in part 
the notion of discrepancy between expectation and reality. 
It could be assumed that the program having been in effect 
for five years should have produced some practices which are 
observable. If such practices are occurring according to 
reasonable expectations » then it could be assumed that 
implementation had, in fact, been achieved. The degree to 
which a discrepancy exists between expectations and reality 
represent the degree to which implementation had not been 
achieved. 

Such a notion requires the development of a set of 
expectations and a means of observing reality to see if those 
expectations were being met. The expectations for implementa- 
tion were essentially a description of those conditions that 

8 For a thorough discussion o£ curriculum evaluation 
models see Sanders and Worthen (in Press, Charles A. Jones, 
Publishing) or P. A. Taylor and D.M. Comley, Readings in Cur - 
riculum Evalaution , W, C. Brown, 197 2 . 



ought to Qxist in classrooms where the program was in 
operation. These conditions would logically follow from a 
careful consideration of the program objectives. Hence the 
first stage of the evaluation model inv/olved (a ) clarifica- 
tion of program objectiv/es and (b) generating of expectant 
conditions. 

The second stage of the model in\/ol\/ed the develop- 
ment of observational instruments. These fell into two 
broad categories--studsnt outcomes and classroom transactions 
and conditions^ The latter category gave rise to develop- 
ment of a general questionnaire designed to assess the degree 
of implementation as seen by teachers and a set of observa- 
tional measures to be used by observers. 

The third and fourth stages of the model obviously 
involved the collection and analysis of the data respectively t 
These stages were to reflect the intent of the evaluation. 
For example, the analysis of the data would focus in part on 
determining where and to what extent discrepancies exist. 
The model was also designed so that the evaluation would 
provide a descriptive picture science teaching in the province 
for the grades involved. Two other adjuncts to the evalua- 
tion model require comment at this point. One involved the 
use of an arbitrary implementation scale and the other the 
validation of measures. 

.In Figure I, a summary is provided of ateps through 
which the evaluation was conducted. 
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' The arbitrary implamentatian ucbIb consisted of a 
set of 28 items huilt into the questionnaire. This scale 
was used in the analysis as a criterion for implementation 
against which to correlate costSp demographic variables and 
other factors of interest. It was also used to stratify 
classrooms in the in-depth portion of the study. 

The other adjunct to the model, built in at every 
opportunity, involved the validation of measures and 
observational schedules. In some cases this was a **boot 
straps* operation while in other situations external criteria 
were sought and employed. In some cases a subsequent stage 
of the evaluation was used to validate a measure used at 
an earlier stage. 

Procedure 

Clarifying of Objectives 

In late December, 1971, and early January, 1972, the 
research team examined the minutes, documents and curriculum 
guides produced by Division III Science Committee for 
Saskatchewan. Any material that was considered pertinent to 
the clarification of objectives was duplicated and distributed 
to members of the Division III Committee, together with 
samples of data gathering instruments for other sources. 

The Division III Science Committee met, together 
with the research team, in early January of 1972. The purpose 



of this meeting was threB-Foldi firstlyi to discuss the 
Division III Science Evaluation project research proposal; 
secondly, to clarify the objectiv/es for the Division III 
Science programp and thirdlyv to operatize these objectives 
in the form of a bank of items which could be used for the 
instruments. Through the use of '•brainstorming" sessions a 
bank of over 200 items were generated. The* process was 
exacting and difficult. It seemed much easier for the group 
io avoid considering objectives and move directly to the 
stating of conditions that ought to exist. 

As a rasul': of this procedure, a criterion emerged 
for program implementation. The items making up the 
criterion were essentially conditions that could be expected 
to exist if the Division III science program had been 
implemented successfully. The criteria can be summarized 
by the follcwingt 

1. Teachers should be u^ing one of the components of 

the Division III instructional package. 

^« Teachers should have access to adequate facilities 

and equipment. 

3. Teachers should be devoting relatively more class 

time to student investigations » student discussion , 
field trips and activities perffiitting students to 
pursue thesir own Interests and lass time to lecturing* 
providing notes and demonstrating. 



4. Teacher evaluation should reflect a greater reliance 

on check llstsf inventories, interviews and student 
observation and less reliance on exams, tests and 
quizzes. 

5« Teachers should have had in-service preparation and 

should perceive themselves as being capable of teach- 
ing the program. 

6« Teachers should have a knowledge of the Division III 

science program and philosophy* 

Instrumentation* 

General Questionnaire 

The items generated by the committee that were suit- 
able for the initial questionnaire were subjected to examina- 
tion for duplication and divided into categories. The 
research team prepared a first draft of the questionnaire 
which was revised on the basis of input from a ^mall panel. 
A second revision of the questionnaire was given to both 
rural and city teachers, to science students and to 
University faculty, in order to validate items and gain 
feedback. As a result of this validation, which involved 
forty subjects, items were modified, subtracted, and added, 
in order to improve the instrument. The third version of 
the questionnaire was validated again using teachers, 
science students, and colleagues, as oiell as the members 
of the Division III Science Committee. Ulhen validating the 

^Copies of the instruments are available upon request 
from marvin F. U!ldeen, Faculty of Education, Simon Fraser 
University, Burnaby 2, 3. C* 



second and third versions of the questionnaire <, the research 
team sought teachers uiho represented various degrees of 
implementation of the Division III Science program. Items 
and groups of items were studied to Insure that the teacher's 
responses reflected their particular situation. In some 
cases the teachers discussed the questionnaire with a member 
of the research team as it was filled out. The resulting 
questionnaire included categories which involved biographical 
information, in-service training, teaching activities, 
teacher feelings, facilities and teacher opinion. 

9 

Arbitrary Implementation Scale . This scale con- 
sisted of 26 items that were judged to be key indicators 
of implementation, and were built into the general question- 
naire. The scale is included as Appendix E with the weight- 
ing and direction for scoring each item. They were selected 
under five categories reflecting the most persuasive elements 
associated with implementation that had grown out of the 
discussions of the Division III Science Committee. These 
catogories Included t 

I. The opportunity and extent of in-service education. 

9 

Throughout the remainder of this report the instru- 
ments will frequently be referred to by their abbreviated 
acronyms. These includes II s Interest Inventory! COG s 
Cognitive neasurei SPSC s Student Perceived Science Class- 
roomi ATT s Attitude Scalei SCOP Science Classroom Observa- 
tion Forms AIS s Arbitrary Implementation Scale. 
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2. The acceptance, knowledge and ??5greement with the 

Division III philosophy. 
3« The self perception of teaching ability for the 

program • 

4« The extent to which certain factors had hindered or 

helped in implementation • 
5« Specific practices in teaching and evaluation. 

The selection of items was made prior to the collection of 
data • 

Student measures 

The instruments administered to the students were 
contained in a question booklet which was reused with the 
students responding on an answer sheet. 

Interest Inventory (II) This measure was developed 
to assess the degree of student preference for science 
oriented tasks or activities. The 34 items included 11 
which were non-*3cience oriented and 23 which were science 
oriented. 

The introductory words for the statements were verbs 
such as learrip collectt writef discuss, be involved, explore 
and plan. They were selected to give some balance to the 
activities and where possible each was used on four occasions, 
three times for science oriented tasks and once for a non«v 
science oriented task. 



IS 

The moa^ura was dev/aloped by genoratinq a larga pool 
of Itama and admini staring thBoi to classes oF qrade 7, 
and 9 students* From this thn ho^t Items uiora selectRd r)n 
the basis of how well each seemed to provide v/btrianca in 
response a^d was related to the students* interest* 

The measure was treated as a Lickert scale and a 
total score generated for each subject* 

Cognit iv/e Weasur e (COG) The 20 items for this test 
were generated from the lifep earth and space science areas 
in grades 7, 6, and 9 respect i\/ely * Every attempt was made 
to construct items so that a correct response required 
thought processes at the comprehension and application levels 
or above* The items were administered to selected classes 
of seventh, eighth and ninth grade students prior to their 
use in the study and revised after items analysis* In 
addition^ the items were critiqued by the Division III 
Science Committee* 

Student's Perceived Science Classroom (SPSC) This 
instrument was developed to provide a measure of the student's 
perception of his classroom* The items made reference to 
activities which typified a classroom where studdEnts ware 
able to utilize materials in conducting investigations, 
discuss results uiith teacher and students » plan some of 
their activities and offer their own ideas* The items were 
reversed to provide a high score where students agreed with 



« 
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the statements. It was assumed that in classrooms where 
students agreed with the statements, the activities des- 
cribed did in fact take place. 

Attitude Scale (ATT) This instrument uias developed 
by Woore and Sutman.^*^ It had a reported reliability of 
•93 and was designed to assess both intellectual and 
emotional attitudes toward science. It is a Lickert type 
scale and provides a single measure of attitude. 

Classroom Observation Schedules 

Observers made use of the Science Classroom Observa- 
tion Form and a set of questions to guide the interviews 
that were conducted with each teacher. 

Science Classroom Observation Form (SCOF) The 
basis of the Division III science program required a class- 
room which was student centered, inquiry oriented and 
provided a setting for manipulative activities. The instru- 
ment, SCOF, was developed in order to provide a rating of 
salient factors operating within this type of classroom. 
Two sources were used in developing the instrument. The 
first involved notions and theoretical constructs prevalent 
in the literature. The second included the specific behaviors 
listed in the Division III Curriculum guide.^^ 



For additional information on this test see R.W. 
noore and R^Y. Sutman. "The Development, Field Test and 
\;alidation of an Inventory of Scientific Attitudes," Journal 
of Research in Science Teaching , 1970 • 7 1 85-94 . 

^^See the Tentative Course Outline Oii/ision III 



Science , September B, 1967. 



Ideally it might be expected that a teacher who had 
implemented the program should act as a guide to learning, 
ask divergent questions and treat science as a tentatiue 
rather than absolute discipline. Emphasis should be giv/en to 
the use of materials as students observe, interpret and 
draut conclusions. The teacher should stimulate and chal-- 
lenge the students in a psychological atmosphere of freedom 
and openness where they interact with materials and each 
other. 

Implicit uiithin such a situation is that materials, 
apparatus, and other learning aids should be available for 
each lesson enabling the student to develop the *'big ideas'* 
and see science as a Mtay of knowing. The student nec^^ds to 
touch, feel, observe, and manipulate materials to provide 
the external referents for internal processes. 

Using these notions and theoretical constructs items 
uiere generated for the instrument representing observable 
manifestations of the theoretical constructs. 

This first draft of SCOP mas circulated to the 
Division III Science Committee, colleagues, and students at 
the Paculty of Educationi Ragina Cairipus for comments and 
suggestions. The revised form was examined, discussed, and 
reviewed by the assembled Division III Science Committee and 
the research team in Regina. This provided for agreement 
of stipulated semantic interpretations of statements and 
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behavior. This was followed by v/isits to two science class- 
rooms where sev/en observ/ers completed a SCOP. The responses 
were compared for each item and each observ/er prov/iding 
some rationale for his responsest This enabled further 
training of observ/ers in consistency of rating and in common 
interpretation of items and behav/ior. Sev/eral items were 
rewritten at this time. 

The scale was arbitrarily divided into three sections: 
environment t students* behavior and teachers' behavior* 
Since the majority of the items were considered interactive 
between the three » the subscales were considered tentative* 

The steps taken in developing the instrument were 
designed to insure face and content validityc The fact that 
the instrument distinguished between classrooms at opposite 
ends of a continuum in terms of implementation during the 
trials provided a measure of criterion-oriented validity. 

SCQF was scored by at least two observers at each 
/observation period and each teacher was observed on at 
least three occasions. The measure was treated as a Lickert 
scale with a total score generated for each subject as well 
as a student » teacher and environment subscale score. 

Interview . At some time during each visitation 
each teacher was interviewed by the team or a member of the 
team. Although this interview was intended to be open-ended, 
a list of 11 questions was used as a guide. 
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Collection of Data 

The general questionnaire was distributed to all 
grade seven, eight and nine science teachers in the Pro\/ince 
of Saskatchevban in February! 1972 via superintendents and 
principals. Of approximately 960 teachers who received the 
questionnaire 9 612 returned a completed version representing 
a 6A% return. The data collected from this portion of the 
study ttias in response to objectives one, two and three 
listed on pages three and four of this paper. 

Tollouiing the return of the general questionnaire 
classrooms uiere selected on a partially random basis to 
insure a sample exhibiting various degrees of implementation. 
The specific steps in the procedure mere as follows s 

1. Each questionnaire uias numbered on the basis of its 
return. From the 612 questionnaires 30 uiere drauin 
using a table of random numbers. 

2. Each of these questionnaires mere examined by three 
people and rated on a one to five scale in terms of 
houi well the respondent had implemented the Division 
III science program. Although the rating uias done 
independently by the three observers, agreement was 
very close in all cases. 

3. The ten cases ranked highest in terms of implementa- 
tion and the ten ranked lowest were then located 
geographically in the province. This location 



involved one classroom uilthin a school. To cut 
douin costs of driving and to utilize the observers' 
time most economically, all grade 7, B, and 9 class- 
rooms uiithin that school uiere observed. In rural 
areas the closest small touin uias picked for a 
similar observation of all Division III classrooms* 
This step uias utilized in order that the ^observers' 
time could be used most efficiently. 
4. The original tuio sets of classrooms uiere then 

ordered at random from 1-lQ . Beginning with the 
first number the superintendent and principal res- 
ponsible for the school in uihich the classroom ujas 
located was contacted and arrangements made for the 
visitation. 

This procedure continued until approximately 40 
classrooms had been obtained. 

The procedures used were designed to strike a com- 
promise between the selection of a stratified random sample 
and the constraints of limited resources. The data obtained 
from the in depth portion of the study was used relative to 
the fourth objective listed on page four. 

Data Analysis 
The results of the questionnaire were considered at 
three levels in keeping with the intent of the evaluation 
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model. The first lev/el was descrip t ii/e involving frequency 
counts on the items and determining means and standard 
deviations. A second level of analysis involved the use 
of chi*square ujhere sets of items mere against other items 
in the questionnaire. These two levels of analysis provided 
a means by which the general picture of Division III science 
teaching could be considered. The relationships detected 
through chi-square were intended to provide some type of 
composite picture of program implementation relative to the 
criteria that had been generated^ The descriptive analysis 
would permit an assessment of the specific areas where dis- 
crepancies existed between the intended and the real situa- 
tion. It was through this data that a profile of a teacher 
who had successfully ioiplemented the program was developed. 
The third level of analysis involved the scoring of the 
arbitrary implementation scale and correalting the results 
with demographic variables on the questionnaire. 

The data from the in depth study analysed at 
through the use of correlational techniques, analysis of 
variance and an attempt to conceptualize the responses to 
the interviews with teachers. 

Student scores on the four criterion measures, 
their response to the activities, grade and sex were inter- 
correlated. This waj done without consideration for the 
degree of p:ngram implementation. In order to obtain a 
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picture of the possible interactions among sex, grade and 
subject preference a 2 x ? x 3 Factorial design was run for 
each of the criterion measures. In this analysis, some sub- 
jects had to be randomly discarded to reduce the ceil size 
to 99 due to limits of the SMD OSV program. 

In order to assess the relationship between the 
scores on SCOP and AIS and the student outcomes, the student 
results were av/eraqed for each classroom and considered along 
with scores on the SCOP measure and AIS scores for each 
teacher. Correlation coefficients were calculated among 
these variables as well as a 2 x 2 factorial analysis of 
variance on the four criterion variables. 

Validation of Measures 

The time devoted to the selection and development of 
the measures was substantial as items were critiqued and 
reviewed at every opportunity and rewritten or reviewed 
wherg appropriate. This was done to insure a high degree 
of face and content validity. 

Since time did not permit an extensive pilot study it 
was considered that for the student measures , additional 
evidence for validity would be provided if certain a^^prior i 
conditions were met by the instruments from the. results of 
the study. 

1. The student measures were selected because they 



were conceptually different# Therefore, high cor- 
relations among them uiere not expectod* 

2. Previous experience with the ATT measure and research 

on attitudes would suggest a low positiv/e correla- 
tion with the scores on the COG measure. 

3« It was expected that the COG scores should increase 

with successive grades • 

4. Students selecting science as a favorite study should 

score higher on the II than those who selected other 
subjects. 

The criteria used to support the criterion-oriented 
validity of the AI5 included the followiriCj: 

1. The AIS should correlate with the teachers* percep- 
tion of how well the program has been implemented. 

2. The AIS scores should show significant differences 
between samples of classrooms using the Division III 
program and the former program* 

3. The AIS scores should corrolate highly with the 
ratings of the research team done on a random 
sample of classrooms* 

Results 

l/alidity of the Instruments 

The a-prior i conditions listed on pages 25 and 25» 
were designed as expectations that the measures should meet in 



order that some claim for their criterion-oriented validity 
could be madet These conditions are discussed relative to 
the data collected in the study. 

The intercorrelations shown in Table 1 are based on 
all subjects in the sample and provide data relative to the 
first two conditions listed on page 26. They are low 

ranging from #06 to .39 indicating that the measures are 
conceptually different. The correlation of #39 betujeen 
scores on the CQC and the ATT measures was within the 
expected range of a low positive, significant correlation. 
The positive, significant relationships that exist between 
the II scores and 5P5C scores as well as the ATT scores and 
SPSC scores were not anticipated. Ulhile the degree of rela- 
tionship, .34 and .32 respectively, is low, it is neverthe- 
less significant and deserves to be noted. It is conceivable 
that those students who showed a high interest in science or 
had a positive attitude toward science would also tend to 
see their classrooms as places where they could use materials 
in conducting investigations, discuss results with teacher 
and student, plan some of their activities and offer their 
own ideas as the two relationships suggest. 

It was expected that the scores from the COG measure 
would increase with each successive grade. This was found 
to be the case as indicated in Table 9 where the scores from 
the COG measure show a significant correlation of .39 with 
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grade. It should be notedt howevert that this correlation 
is based on the classroom naans rather than individual 
student scores. The value of the correlation indicates 
that the means of the eighth and ninth grade classrooms 
ttiera successively higher than the seventh grade classrooms. 

Relative to the fourth expectation the sample of 
students was split into those who indicated that science 
uias their first or second chc Ice of subjects and those tuho 
selected other subjects. Mhan compared on the basis of their 
means on the II the groups differed utidely. Thone favor- 
ing science had a mean of 117.25 and the other group 105. 55. 
This difference which was significant at the .01 level was 
approximately one standard deviation. Ulhile this item of 
information seems to support the obvious* it does neverthe- 
less add something to the validity of the II measure. 

Since the use of AI3 represented a crucial stage of 
the evaluation model additional steps were taken to gather 
evidence oncerning its validity in keeping with the criteria 
listed on page 26. 

Data from the general questionnaire provided informa- 
tion relative to the first and second criteria. The res* 
ponses to items 38* 39 • and 40 which asked teachers in the 
three grades to state their opinion on the degree to which 
the Division III program had bean implemantad. were correlated 
with scores on the AIS. The correlations were significant 



Table 1 



Intercorralations Among Student measures, 
Sex and Grade with Reliabilities-f 
for Student Measures Shown 
in the Diagonal 



Variable 




Student 


Measures 




II 


SPSC 


Air 


COG 


Student Rsasures 












II 


.73 










SPSC 


.34* 


.87 








ATT 


.27 


.32 




.71 


.56 


COG 


.13 


.06 




.39«- 




Sex 


-.34* 


-.07 




.01 


-.07 



♦ The reliabilities reported represent internal 
consistency except in the case of the COG 
Mhich is KR-20 reliability. 



• Significant at th« .05 level. 
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with values of #53, .A7 and .3A respectiv/Qly for seventh, 
eightVi and ninth grade teachers* 

The second criteria stated that the AIS should shouc 
differences between different groups in the sample, i^^eans 
for different groups classified according to grade and pro- 
grams taught are provided in Table 2. It can ba seen that 
differences do exist. For example, ♦'Life science 
group are approximately half a standard deviation above the 
••Earth 6^ group. This was expected since evidence pointed 
to a better implementation picture in grade seven than the 
eighth and ninth grade. TBachers of the •'IPS" group, a 
highly laboratory oriented program, exceeded all others by 
nearly one standard deviation. Of most significance is 
the mean of the ** 5cience Activities 7 and 6 '* which is well 
below all other groups* 

The data provided in Table 3 is related to the 
third and fourth criteria* As indicated earlier the three 
member research team independently ranked a random sample 
of completed questionnaires according to degree of imple- 
mentation. This was done on the basis of a subjective 
study of each questionnaire without prior consultation as 
to prior criteria for ranking* or specific reference to 
the AIS it^msm The AIS scores from these questionnaires 
were later determined for the upper and lower ten. The 
mean difference between these groups as shown in Table 3 is 
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msan Ualues of AIS Scores for Different 
Groups in the Study 



Croup 




Hi CO II 




Grade 7 


126 


109.58 


15.76 


Grade 8 


79 


105.05 


14.43 


Grade 9 


124 


106.69 


13.95 


Grade 7 4 8 


145 


108.10 


15.77 


Grade 8 & 9 


36 


109.19 


17.3? 


Grades 7, 8 & 9 


2n 


110.54 


16.56 


Life science 7 


152 


112.28 


15.19 


Life 4 aarth 7 4 8 


143 


110.07 


15. ?R 


Earth 8 


54 


103.69 


14.60 


Space 9 


74 


104.36 


12.0? 


modern Earth 8 4 9 


19 


107.80 


14.73 


Science Activities 7 4 8 


17 


97.33 


12.63 


IPS 


27 


121.15 


12.89 
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highly significant. 

Thfi data for the second comparison shown in Table 3 
was obtained by selecting a school system where variation 
in program implementation existed* Tha superintendent tuas 
then asked to indicate the programs used by the teachers. 
Those teachers using the Science Activ/it ies were compared 
uith those using Division III programs on the basis of 
the mean scores from AIS. It can be seen that the diiffer- 
ence between the two groups is significant. 

The reliability of the measures are presented in 
Table 1 in the case of the student measures and Table 7 for 
the AIS and SCOP. Utith the possible exception of the COG 
msasuret all others would appear adequate for purposes of 
this study in which group comparisons were being made. 

On the basis of the steps taken during the develop- 
ment of the measures and the results reported in this sec- 
tiont the investigators feel confident in stating that tha 
measures have distinct conceptual underpinnings as indicated 
by low intercorrelationSf possess a reasonable degree of 
validity (measure what they purport to measure) and are 
sufficiently reliable (measure Cfonsistently ) for purposes 
of thi§ study. 

Implementation 

The first objective of the study has been to assess 
the extent to which local school districts had adopted the 

ERLC 
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instructional package and conductod tli*? appropriate in-ser\/ic:e 
education program. Adoption of the instructional package 
was intended, in an ideal sense, to satisfy the criteria for 
program implementation (summarized on page l4-15XTh9 results 
considered relative to this objective included responses 0*0 
the questionnaire , teacher comments , ^nd the relationships 
detected through the use of chi-square. The criteria are 
repeated here to facilitate discussion. 

criterion t Teachers should be using one of the components 
of the Division III instructional package. 

Retttieen BS% and 9Q% of the teachers uuere using one 
or more components of the suggested package. 
criterion t Teachers should ha\/e access to adequate 
facilities and equipment. 

Fifty-five per cent of the classrooms were opnraling 
with minimum facilities for science teaohlncj, Thn .ivi^raq** 
teacher reported hav Inq acce**R to approx Ima i n 1 y ^?lin . flfl nf 
equipment for the teaching program, 

criterion t Teachers should be devoting relatively more 
class time to student investigation 3, student discussion, 
field trips and activities permitting students to pursue 
their own interests and less time to lecturing, providing 
note and demonstrating. 



criterion t Teacher ov/aluation tshould r«Flect a greater 
reliance on check listst Inventories, interviews and 
student observation and less reliance on axamins, tests and 
quizzes. 

The amount of class time devoted to teaching 
activities and the emphasis given different aspects of 
evaluation aw ranked ii^BidivbtezA-ac cor ding to their mean 
values. Student investigations rank high in the class 
activities along uiith lecturing and demonstrating and dis- 
cussions. Field trip activities and pursuing interests 
rank lowest. Quizzes and tests are the most predominant 
form of evaluation while interviews rank lotwest in this 
regard. 

When these rankings mere compared across different 
components of the instructional package the order remained 
approximately the same but the mean values differed markedly. 
The emphasis placed on different evaluation activities 
showed a more varied pattern for the different instructional 
components than was the case for instructional activities. 
criteri on t Teachers should have had in-service preparation 
and should perceive themselves as being capable of teach- 
ing the program. 

Results indicated that only 51>J of the respondents 
had had an opportunity for in-service education. Forty- 
eight per cent reported that science was not the subject that 
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they were most capable of teaching. 

criterion » Teachers should hav/e a knoiwladge of the Division 
III service program and some understanding of its philosophy. 

Data relative to this criterion were gathered from 
items on the questionnaire r comments made by teachers as 
ujell as the interviews. Ulhile teachers seemed generally 
auiare that a Division III program existed there were large 
numbers who exibited substantial ignorance as to its nature 
and philosophy. 

Factors Critical in Implementation ^ 

The second objective of the study was to relate 
implementation to such factors as the costs for equipment, 
the science background of teachers, the amount of in-service 
and facilities. This was done by correlating responses on 
these variables to the scores on an arbitrary implementation 
scale. Table 4 summarizes these results. 

It can be seen that costs for equipmenty science 
teaching capability and the availability of in-service show 
the highest relationship with scores from the arbitrary 
implementation scale. Age and teaching show negligible 
correlations. The amount of academic science and avail- 
ability of facilities are also positively correlated with 
implementation . 

The third objective of the study was to obtain a 
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Tab 1b 4 

Correlations Between Arbitrary Implementation 
Scale and Selected Variables 

/ 



Variable 


Correlation uuith 
Arbitrary Imple- 
mentation Scale 


Sax 


• 06 


Age 


.01 


University Education 


.22* 


Academic Science 


.31** 


Teaching Experience 


.06 


Science Teaching Capability+ 


.50** 


In-service Education (Opportunity to 

attend) 


.47** 


Number of Science Classes Taught 


.24* 


Facilities 


.37** 


Costs for Equipment - Grade 7 


.57** 


- Grade 8 


.49** 


- Grade 9 


.40** 



* < .05 
** < .10 

+ Response to an item asking if science was the subject 
they were most capable of teaching. 
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profile of ch ira*: u^r i '...s , bi^'. h :it^nH.iqrar:tiical und 

o r q a r : i z a t i 0 n a 1 , t y p i ; : a 1 i; f t j h i h i n\ p 1 r ] ; n w j 1 1 a i d n . 1.3 v e r a 1 

sources of dana \k'.ev9. used in arriv/iiig lit H;is prnfllo 

inc luding the correlational ana lysis dlscjjsjsd proviausiy ^ 

teacher rnmrnan'-^s , In ter vie'^is and the relal ionshipa detected 

among variables i^hrouqh the chi-^squar^j an'c:ilysi3. 

Univ/er jity eiuc at ion and acadGml c hours of ^ciencs 
appeared to be L>tr. ny facto:::- in impiainentation » snaring 
many parallel reialionships* Thos^td \;r.zh more uni\/8rsity 
education and academic houro of ocL^^ncB rnported po3iti\/e 
perceptions uuith rtiqard to their carjab ilit ies » pra-sarv/ice 
and .in-ser\/ice enucatiori, philosophy and kno'jiiledge of the 
program. Thay reported usinrj a "arxc-ity ^.nd mi;:tur2 of 
learning and e\/aluati\/8 actii/itios th^t indicated inf)plementa- 
tion. Those .AJilh loss university education exhibited 
opposite trends r notably using fewer student in\/estigat ions 
and having a pa 1 1 srn of ac L iuiu iv;s uih •] ch indicated lo^ju 
implementation of thtj program. 

TeacherTj in irn.'in ara'ds r;"^:;:G3:* tnd a pattern of res- 
ponses sugger>r.inn a nigher l^ual of inu:- 1 emc^n ta t ion than 
thuir rural cc:on L ar . »ar ts * The excap Vion to this was that 
the retention of cid'-^r programs vsaa !":i':;her in large cities 
than rural areas. ..rnaller centres ri-oartsci ?j better 
implementation of the grade 7 life acic^nci^ program. 

Older teachers tended to be slightly ^nore "formal" 



with their activitlesp utilizing mora uirltten reports and 
notes taking, besides feeling they had to stick closely to 
the subject matterp despite attending more in*ser\/lce 
sessions than the younger teachers. 

Those ufho utere teaching several classes in science 
perceived themselves more capable of teaching the subjecl, 
had attended more ln*-servlce sessions and generally might 
be considered as having more success with the program than 
those teaching one class. 

On the general questionnaire teachers were asked to 
rate on a 5 point scale eight factors in terms of their 
usefulness in helping to implement the program and how much 
they had hindered impiementat ion . These factors are ranked 
according to their means in Table 5, It can be seen that 
administrative and school board support, favorable student 
reaction and belief in philosophy rank high in terms of 
usefulness in implementation. However, these four are the 
lowest factors in terms of hindering implementation. Lack 
of equipment is ranked highest in this regard. 

From these results it was possible to construct a 
profile of factors reported by teacher s associated with a 
relatively high level of implementation. They were more 
likely working in elementary or junior high schools, 
teaching the seventh grade life science program, and teach- 
ing science to more than one class. They typically had more 
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^ V 

years of uniueroit/ and dcad^n-iic acience vuith ^^ to IC yp.Tir^: 
of teaching experience, ^ucn teacners ujex'-a inore likely 
have attended in-service sessions, to be working in class- 
rooms with good space facilities and ecuipment, and perceiv- 
ing admiinistrative support as being useful. They had a 
high understanding of philosophy and objectives, a high 
knowledge of content and a strong agi eenent with the 
philosophy of the program. 

The teachers who revealed the louiest level of 
implementation exhibited a profile opposite to that de.^i- 
cribed above. In particular, they taught in a rural area, 
had feujer hours of academic science, taught science to fewer 
classes and perceived administrative support as a hindrance. 

Program Use and Student Outcomes 

The fourth objective of the study was to compare 
the student outcomes for those classrooms that had implemented 
the program with those that had not in order to relate 
student outcomes to implementation. The scores on the AI5 
and SCOF were used as measures of classroom implementation. 

The data in Tables 6 and 7 provide pertinent informa- 
tion. It should be noted that the correlations in Table 5 
were based on all 43 classrooms involved in the study with 
pairwise deletion for missing data. The data in Table 7 is 
based on 30 classrooms because those cases having missing 
data were excluded from the analyses. 
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From Table 6 it can bo soen -Ma - only in ti e car,^ 
of the 3PSC does a pattern of significant rclacionuhios 
emerge uiith SCOr and AI5« Here correlations cf .31^ and 
•39 exist between the SPSC scores and the pupil, teacher 
and total scales of SCOF respectively • Also a significant 
relation exists beti^aen the AIS scores -^nd the scores on 
SPSC. The II scores bear a relationship to only the pupil 
subscalo. The ATT and COG scores art unrelated to the 
SCOr and AIS scores. 

The analysis of variance using SCOF and AIS as main 
effects and the student measures as dependent variables is 
shown in Table ?• The analysis is complementary to the 
previous correlation analysis but it provided hero to detsr* 
mine if there were interactive effects betojfifcn the AI5 ?jnri 
SCOr main effects. 

It can be seen that in only one case is a F value 
significant. This was where the SPSC scares ujere dependent 
variables and AIS the main effect. An examination of the 
cell means indicated an evaluation of scores on the SPSC for 
the top AIS group. No other F values mere significant 
indicating no effects for the other measures and no inter- 
active effects. 

In order to control for grade three other analysis 
mete conducted. The AIS scores and grade level were used 
as main effects in one analyses and SCCF scores and grade 



level in another* Tuio analyr>DS wcrr run because nr^t-, 
cells resulted when a 2 x 2 x 3 analysis was conductua. \ 
third analysis was run using the 2x2 uasion usad above 
with grade level a& a co-variate« ko oiffcrances werts 
revealed in these analyses that had noi been revnalej 
previously* 

Discussion 

It was found that while over B5% of the schools 
had adopted the curriculum package, classroom practicw 
and evaluation techniques had not changed substantially 
from what might have been expected prior to the introduc- 
tion of the program. Also, it was found that the implementa- 
tion phase of the program had broken down most significantly 
in the area of in-service education among teachers. These 
two results are linked and suggest the obvious conclusion 
that substantial educational changes can not be brought 
about through legislation , fiat , or widespread adoption 
that is not coupled with intense and prolonged in-service 
education. 

Another result had implications related to this 
conclusion. It was found that teachers with six to ton 
years experience were having most success vjith the procram. 
These were teachers who were entering or had just entered 

the teaching force when the program was being introduced 



4 ■ 

into tha schoolr.. This corresponds with ^ neriod of fairlv 
intensive in-service t : aining when work hops »Aiero heir: ; 
held concerning th^j program and when tQacners i^iere bf2i~g 
asked to try new programs. This coul;> U3 taken iio evidcn*. o 
that in-service education does have an effect upon thn ionq 
range practices of teachers* In addition, it underlineii 
the need for in-service activities to ba carried out over rti 
extended period following program implQmentation. 

A main purpose of the study ujas to compore student 
outcomes for classrooms in ujhich the program had been 
implemented with those where it had not. fhs results 
indicate that in only one case was a significant difference 
found. This difference occurred in Ihe area of classronm 
perception where it was found that in those classrooms ujherfj 
a higher degree of implementation had occurred the students 
perceived themselves better able to utilize materials in 
conducting investigations, to plan their own activitios and 
to offer their own ideas. None of the other criterion 
variables were found to be significantly different for the 
two groups* 

These results might be interpreted as another 
example of the "no significant differences"* pattern so 
typical in education. However 9 the Interpretation would 
seem to go further to suggest that (a) fundamental and 
widespread changes in classroom teaching are unlikely 
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through this ' yn»» o** program dr^^/n ' on -i^nt :md impl d vt^-f* ^: ic^^ ♦ 
(b) lev/els at which changa in educc^ticn can ba dotecL??:! 
uiould seem to exj^st. 

The first of these interpretations requires liltl- 
elaboration exjcept to reiterate that the study did net 
reveal substantial changes in teacher behavior and proc- 
tices of evaluation, and then one should not expBCt dif- 
ferences In studr?nt outcomes • 

In the cace of the second interpretation, there 
uiould appear to be implications for future research in 
curriculum cnange. The differences in student outcomes 
revealed in this study occurred at the level of a student^s 
perception of his or her classroom. A subsequent level 
might be the interests that students deve^iop and further 
level their attitude and cognitive ability. Thus a hier- 
archy might ttiall exist, the lotuest rung of which, is the 
student perception of his classroom. Hence when attempting 
to identify the effects of a program this level might 
well provide an initial indicator* 

The results of the study both directly from inter- 
views and implied from other data pointed to several con- 
straints to effective teaching and program change. 

It would appear that the constraints on teachers 
and their perception of how curriculum reform should cotrie 
about are diametrically opposed* Mone of the teachers 
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interviewed indicated ^ desire to hav'e curriculuii change 

emanating from an outside group. Yet tho constraints they 

outlined on their nPfectiye teaching and the lack of 

professional interchange suemed to leav/e little opportunity 

for curriculum development. In short., if a teacher is too 

busy to do an effective job of teaching, how can he be 

expected to work on an improved program of his students. 

The sex differences for interest were very clear in 

12 

this study and are consistent with other findings. The 
implications remain obvious for thosG interested in cur- 
riculum development and for those concerned with a science 
program designed for the general population of students. 
The context of the courses and the predominance of male 
teachers must surely represent the most likely causes of 
this type of finding. However, it would seem that further 
research is needeo to pin down the precise reasons for these 
differences and that future curriculum planning should take 
this result into account. 

The profile of factors associated with a relatively 
high level of implementation was as follows t working as 
an elementary or junior-high teacher i teaching science to 
more than one class r having had more years of university 
education and more hours of academic science; having three 
to ten years of teaching experience; having attended 

12 See M.F. Wideen , "Comparison of Student Outcomes 
For Science - A Process Approach and 'iraditional Science Teaching; 
Foi Tnjrd, Fourtli, Fifth cxnd "Sixtli Grade lI Jesses: A Product ' 
Evaluation**" tin Vress, Journal ot" Research in Science Teaching ) 
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in-service sessions and found thcr* Uotful; indicbt.-n: 
having a high understanding of phMc of.^/ anc' objecti /e^; 
indicated a high knowledge of contL'nt> mateirials, ann 
equipment; having a high agraement u^ti^ m^i losop hy , i^or*^- 
ing in classrooms with good spaco r.^cilitics anc : c ii:*n^ ^ 
perceiving administrative support as 'ceing jsiLf ul • 

The teachers showing the loiiast Isvei of impl^^enta- 
tion exhibited a profile opposite to that described abovoo 
The strongest of these included the follouinQ: teaching in 
a rural area; having fewer hours of academic science; teacn- 
ing in a rural axeaj having fewer hours of academic science; 
teaching science to fewer classes; and perceiving administra- 
tive support as a hindrance. 

The two most successful programs in the packaQe 
appeared to be the grade 7 life science program and th^:; ITL 
program in grade 9. The tuio bear little resemblance except- 
that they are both highly laboratory oriented. They differ 
in context, the amount of structure involved and in the 
manner in which they were developed. The IPS program was 
developed as a curriculum package in the United States 
and in a highly organized program centering around a single 
conceptual scheme. The life science program was developed 
within the provincot and lends itself to diversity. 

The success of the two may be due in part to the 
committment of the teachers using them. Tho life science 



program is taught by teachers 'juho rr^- Inn fir:;': tin- 
involved in a Iciljoratory teaching ni'"uacion, wH i r : / ^ 
equipment to uiork with and are facing on^^-huiiiastic stu j -', . . 
These factors would surely combina to produce highly 
committed and motivated teacher. The I.'w t^^.ichc avo 
using a program not recommended by the Depdrtment of '-I&ac-^-- 
tion. Therefore to use it, it might be assurn^d that they 
have a strong comnitmenc to make it uiork . 

The concapt of commitment may also tie a cause for 
the rather poor l^vel of itTiplementat ion and succeso of cfifi 
space science program and to a lesser degree the earth 
science program. In having had the program introduced 
throughout the province many teachers may not be cDmrnitted 
to its value* 

The notion of commitment to a program and its 
success would appear to be a fruitful areri for further 
study. If it can be shoiwn that the level of commitment 
is a crucial factor in determining & program's success, 
then numerous implications exist for program development. 

In conducting the evaluation, the investigators 
were constantly aware of the constraints imposed by the 
absence of context and input which should have been 
available from earlier stages of program development. Such 
data would have provided a basis with which to compare 
the results but more importantly, a context from which to 
raise more pertinent questions. It would have also aided 



in the sal ecti on of programs . Thu^i Lhe process of proccje 
ing with program dBvolopment prior tn oaginning the proco 
of evaluation and data gathering 'm-s little to recommsnd 
it. 

A further constraint upon Ine evaluators uias thi? 
lack of guidelines and curriculum models from which tney 
could draw. The theoretical models were largely inap- 
propriate since they remained theoretical and little 
published data uiare not available on their ef f sctivsnass 
or use. The suitable strategies to employ in a post hoc 
situation faced in this study, were not available through 
the literature. Thus the approach or model used in this 
evaluation which made use of an implication scale is 
offered a means others may wish to consider. 



